extended_walkforward_111day - README

Duelling Deep Q-Network Trading System
Comparison of BASE and VWAP State Representations using Walk-Forward Trading Scenarios

---

Project Description

This project presents an algorithmic trading system based on Deep Reinforcement Learning. The proposed framework employs a Duelling Deep Q-Network (Duelling DQN) to learn trading strategies from intraday financial market data.

The primary objective of the project is to investigate whether enriching the agent's state representation with Volume Weighted Average Price (VWAP) features improves its trading performance compared with a baseline representation that relies on conventional market indicators.

To ensure a fair comparison, the BASE and VWAP configurations share exactly the same:

trading environment,
reward function,
learning algorithm,
neural network architecture,
training procedure.

The only difference between the two configurations is the information available to the agent through its state representation.

---

Algorithm

The trading agent is implemented using a Duelling Deep Q-Network (Duelling DQN) architecture.

The neural network consists of two separate streams:

Value Stream, estimating the value of the current state V(s)
Advantage Stream, estimating the advantage of each available action A(s,a)

The final Q-value is computed as:

Q(s,a)=V(s)+(A(s,a)−mean(A(s,∗)))

This implementation uses the standard Dueling DQN architecture.

---

Dataset

The experiments use minute-level historical market data for the following stocks:

AES
AMD
CHK
F

The complete dataset contains 111 consecutive trading days, providing realistic intraday price and trading-volume information.

---

Experimental Scenarios

The project contains two complementary experimental scenarios.

1. Pilot Walk-Forward Scenario (6-Day Dataset)

This scenario is primarily intended to:

verify the correct implementation of the complete trading system,
validate the interaction between the agent and the environment,
provide an initial comparison between the BASE and VWAP configurations.

The scenario uses six consecutive trading days.

One experimental episode consists of five consecutive Train → Trade cycles:

Day 1 → Day 2

Day 2 → Day 3

Day 3 → Day 4

Day 4 → Day 5

Day 5 → Day 6

The portfolio is initialized once with 100,000 USD and evolves continuously throughout the five trading days.

The final portfolio value corresponds to the result obtained after the fifth trading day.

2. Extended Walk-Forward Scenario (111-Day Dataset)

The second scenario constitutes the main experimental framework of the study.

The complete dataset consists of 111 trading days, which are processed through multiple independent walk-forward episodes.

Each episode:

starts with an initial portfolio value of 100,000 USD,
contains five consecutive Train → Trade cycles,
finishes with all open positions closed,
resets the environment before the next episode begins.

This design allows independent comparison between successive walk-forward episodes while preventing information leakage across different evaluation periods.

---

Train → Trade Methodology

Both experimental scenarios employ the same learning strategy.

For every cycle:

Train Day (D)

The agent learns using historical market data.

↓

Trade Day (D+1)

The learned policy is applied to the following trading day without further parameter updates.

This methodology avoids look-ahead bias and provides a realistic simulation of sequential trading decisions.

---

BASE versus VWAP State Representation

Two different state representations are evaluated.

BASE

The agent receives:

normalized market features,
technical indicators,
portfolio variables,
position-related variables.
VWAP

The VWAP configuration includes every BASE feature together with additional VWAP-derived variables.

VWAP is computed as

VWAP=
∑volume
∑(price×volume)
	​


The VWAP configuration modifies only the state representation.

It does not modify:

the trading execution mechanism,
the reward function,
the Dueling DQN architecture,
the learning algorithm,
the training procedure.

This enables a controlled experimental comparison of the informational contribution of VWAP.

---

Reward Function

The reward function combines profitability with risk-aware performance measures.

The base reward is derived from realized portfolio performance and transaction costs.

Additional reward shaping incorporates:

Sortino Ratio
Drawdown Penalty

The final reward therefore encourages the agent not only to maximize profitability but also to control downside risk and excessive portfolio drawdowns.

The reward formulation is identical for both BASE and VWAP configurations.

---

Trading Environment

The trading environment simulates realistic intraday market behaviour.

At every time step the agent receives:

current market information,
portfolio information,

and selects one of three possible actions:

Hold
Buy
Sell

The environment continuously updates:

portfolio value,
open positions,
transaction costs,
exposure statistics,
reward.

---

Output

The system automatically generates CSV files containing detailed execution statistics, including:

portfolio value,
daily and cumulative rewards,
trading activity,
long and short transactions,
transaction costs,
exposure statistics,
traded volume.

These files are subsequently used to generate the figures and quantitative analyses presented in the accompanying thesis.

---

Project Objective

The objectives of this project are:

develop a Dueling DQN trading agent,
compare BASE and VWAP state representations,
evaluate trading performance under two walk-forward scenarios,
investigate the contribution of VWAP-derived information to the agent's decision-making process.

---

Main Finding

The experiments indicate that enriching the state representation with VWAP-derived features modifies the trading behaviour of the reinforcement learning agent but does not consistently improve overall trading performance.

The results highlight that adding new input features to a Deep Reinforcement Learning system does not automatically lead to better performance; rather, their effectiveness depends on the informational value they provide beyond the existing state representation.